Fully Automated Cardiac Shape Modeling

End-to-End automation of the Cardiac Atlas Project, from SSFP MRI to CIM Mesh.

Steps:

  1. View selection
    • Export Slice Info File
  2. Short axis slice selection
  3. End-diastole and end-systole selection
  4. Landmark localization
  5. Segmentation
  6. Guide point extraction
    • Export guide point file

If your device has a gpu and the necessary software installed, tensorflow will be able to detect it and display it here. If not, the entire notebook can be run on the CPU, but will take a significantly longer amount of time, particularly if normalization and test time augmentations are utilized during landmark localization.

# Only use CPU for right now - Set CPU as only available physical device my_devices = tf.config.experimental.list_physical_devices(device_type='CPU') tf.config.experimental.set_visible_devices(devices= my_devices, device_type='CPU')

Step 1: View Selection

The automated view selection is designed to automatically identify and select MRI views that are relavent to cardiac modeling. The analysis is designed to be run over a directory containing one or multiple patient subdirectories. The code runs the analysis iteratively over each subdirectory.

The suggested directory structures for each approach are shown below:

Run the analysis over each patient in a directory individually (i.e., analysis for patient 1, then analysis for patient 2.)

├── DATA
    ├── Patient1
    │   ├── 1.dcm
    │   ├── 2.dcm 
    │   ├── 3.dcm          
    │   └── ...   
    └── Patient2
        ├── 1.dcm
        ├── 2.dcm 
        ├── 3.dcm          
        └── ...

ENTER USER SETTINGS

Change the settings below to match the desired patient and correct src, dst, and csv_path.

Useability Note

If necessary, the predicted views can be corrected using the code below (currently set to Raw NBConvert) in order to limit accidental changes. Enter the series number to change, the currently predicted view, and the actual (or correct) view. The code will copy the dicom files to the correct directory.

Useability Note

The number of frames (or phases) per slice is automatically selected from the SAX stack. In some cases, the number of frames per slice may be different for different views (e.g., 30 frames per slice in the SAX, but only 20 frames per slice in the 4CH view). If this is the case, be sure to appropriately enter change the number of phases when performing landmark localization predictions below.

Similarly, if a dicom file is missing, the number of frames per slice may be inaccurately low. For example, with one dicom missing, the number of frames might be set to 29, when it should actually be 30. Be sure to manually correct this error if it occurs.

# fix directory structure if necessary for series that were incorrectly predicted series_num = 12 pred_view = 'OTHER' actual_view = 'RVOT' if not os.path.isdir(os.path.join(dst, patient, actual_view)): os.mkdir(os.path.join(dst, patient, actual_view)) for file in os.listdir(os.path.join(dst, patient, pred_view)): if '.dcm' in file: dcm = pydicom.dcmread(os.path.join(dst, patient, pred_view, file)) if dcm.SeriesNumber == series_num: shutil.move(os.path.join(dst, patient, pred_view, file), os.path.join(dst, patient, actual_view, file)) if len(os.listdir(os.path.join(dst, patient, pred_view))) == 0: os.rmdir(os.path.join(dst, patient, pred_view)) print('Done!')

Export 1 - Generate Slice Info File

Now that the views have been selected, we can export the slice info file for use with the BiV Modelling v2 code. The first code generates a pandas dataframe of this information (used throughout this notebook) while the second code exports it to a txt file.

Step 2: Phase Selection - ED and ES

The following code performs end-systolic phase selection from the short-axis stack of images selected above. The end-systolic phase is assumed to be 0 (may not necessarily be the case in all scenarios, but has been for all of the cases processed to date).

The model performs an ES phase prediction over each short-axis slice, then averages the predictions to produce a final result.

Step 3: SA Slice Selection

In the next step, we will select the slices of the SA stack that will be useable for cardiac modeling (typically spanning from the apex to the base).

To improve consistency of the validation between the automated pipeline and the manually generated models, I also incorporated code to automatically load the slice info file from manually generated cases, and display which short-axis slices were selected. If you do not have the manual slice info files for your cases, simply skip this cell block.

The code below organizes the predictions made by the slice selection model, and incorporates manual labels if available.

Useability Note

Manually selected SAX slices typically range from base to apex, with every other slice selected. However, both the segmentation and landmark localization models struggle on the most apical and basal slices. Consequently, these slices are optimally excluded from the automated pipeline. In almost all cases I processed, the most basal manual SAX slice was too far basal for use in the automated pipeline. Basal slices that transect the valve planes and/or right ventricular outflow tract should be excluded.

I typically select 3 to 4 SAX slices, ranging from just below the valve planes to the apex. I select every other slice, similar to the manual cases.

Step 4: Landmark Localization

The following code can be used to perform landmark localization. The points that are predicted are the following:

Useability Note

The RV insertion points from the SAX stack are typically fairly accurate; consequently, normalization and test time augmentations (both of which significantly increase required processing time) may not be necessary. For the 4CH, 3CH, and RVOT views, I recommend including both normalization and at least 5 test time augmentations.

4.1 Short-Axis RV Inserts

Predict the RV insertion points from the SAX stack.

4.2 Four Chamber Landmark Localization

Predict the MV inserts, TV inserts, and LV apex from the 4CH view. Select only one 4CH slice for inclusion in the final model.

Useability Note

For the landmark localization models, cardiac orientation matters. These models were trained on images with the LV apex pointed towards the top right corner of the image, with the RV (and tricuspid valve) located above the LV (the standard orientation). If this is not the case for your images, utilizing the provided settings "flip_ud" (flip image up/down) and "flip_lr" (flip image left/right) can be used to achieve the correct orientation. This is necessary to achieve useable predictions if your orientation is incorrect.

Useability Note

If the landmark localization predictions are incorrect, they can be manually corrected using the Annotate module. Ideally, this tool will be improved in future versions of this notebook.

Referencing the landmark dataframe above, enter the index number (0 or 1), slice ID, and landmarks to correct. The Annotate class will open a matplotlib figure of the view / slice ID selected. Draw a line on the image across the valve plane by clicking twice, once at each valve insertion point. Once the line has been drawn, save the correct valve insertion points by running annotator.update() in the cell block below. The 4CH views are typically accurate; however, the AV and PV insertion points below may need to be corrected.

# if necessary, update landmark predictions from annotations import Annotate # Settings index = 1 slice_id = 11 landmarks_to_correct = ['TV1', 'TV2'] # should be list of labels (e.g., ['AV1', 'AV2']) annotator = Annotate(four_chamber_df, volume, index, slice_id, landmarks_to_correct) %matplotlib notebook annotator.plot()# update dataframe and display four_chamber_df = annotator.update() four_chamber_df

4.3 Three Chamber Landmark Localization

Predict MV insertions and AV insertions from the 3CH view.

4.4 RVOT Landmark Localization

Predicte PV insertion points from the RVOT view

If necessary, update the PV insertion point predictions.

from annotations import Annotate # if necessary, update landmark predictions # Settings index = 1 slice_id = 0 labels = ['PV1', 'PV2'] # should be list of labels (e.g., ['AV1', 'AV2']) annotator = Annotate(rvot_df, volume, index, slice_id, labels) %matplotlib notebook annotator.plot()# update dataframe and display rvot_df = annotator.update() rvot_df

4.5 Select 2CH Views

We do not include landmark points from the two-chamber views; however, it is still optimal to include the segmentations from these views in the final model. If available, select one two-chamber left and one two-chamber right view for inclusion in the model.

5. Segmentation

Myocardial segmentation is performed using nnUnet. Double check that the input and output folders are correct below.

For use with the nnUnet pipeline (the model used for segmentation), we need to convert each of the relevant image files to a NIFTI file. The below code used Simple ITK to accomplish this task.

In the below code, we will iterate through each of the cardiac views and use the appropriate nnUnet model to make predictions. Models are saved under models/Segmentation/ and are organized by task (i.e., view).

ENTER USER SETTINGS

Change the model folder to the absolute path for the segmentation models (e.g., "/home/ubuntu/CAP/CAP-FullAutomation/models/Segmentation").

6. Extract Guide Points from Segmentations and Landmarks

Processes the segmentations and landmarks by extracting contours for inclusion in the guide point file. As a part of this process, we perform an inverse transform from image to model coordinates using the affine matrix provided in the DICOM header. Landmark points are loaded from the csv file that was saved previously. To display guide points as they are generated, set display = True in the .extract_guidepoints() function.

ENTER USER SETTINGS

Provide the absolute path to the image (with .nii inputs), segmentation (with .nii segmentations from nnUnet), and final output folder.